{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CCQY7jpBfMur"
      },
      "source": [
        "##### Copyright 2018 The TensorFlow Authors."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "z6X9omPnfO_h"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F1xIRPtY0E1w"
      },
      "source": [
        "# Keras"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VyOjQZHhZxaA"
      },
      "source": [
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/r1/guide/keras.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://github.com/tensorflow/docs/blob/master/site/en/r1/guide/keras.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BPe-206xGPfe"
      },
      "source": [
        "> Note: This is an archived TF1 notebook. These are configured\n",
        "to run in TF2's \n",
        "[compatibility mode](https://www.tensorflow.org/guide/migrate)\n",
        "but will run in TF1 as well. To use TF1 in Colab, use the\n",
        "[%tensorflow_version 1.x](https://colab.research.google.com/notebooks/tensorflow_version.ipynb)\n",
        "magic."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ViAXWoKlZ8s6"
      },
      "source": [
        "Keras is a high-level API to build and train deep learning models. It's used for\n",
        "fast prototyping, advanced research, and production, with three key advantages:\n",
        "\n",
        "- *User friendly*<br>\n",
        "  Keras has a simple, consistent interface optimized for common use cases. It\n",
        "  provides clear and actionable feedback for user errors.\n",
        "- *Modular and composable*<br>\n",
        "  Keras models are made by connecting configurable building blocks together,\n",
        "  with few restrictions.\n",
        "- *Easy to extend*<br> Write custom building blocks to express new ideas for\n",
        "  research. Create new layers, loss functions, and develop state-of-the-art\n",
        "  models."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IsK5aF2xZ-40"
      },
      "source": [
        "## Import tf.keras\n",
        "\n",
        "`tf.keras` is TensorFlow's implementation of the\n",
        "[Keras API specification](https://keras.io). This is a high-level\n",
        "API to build and train models that includes first-class support for\n",
        "TensorFlow-specific functionality, such as [eager execution](#eager_execution),\n",
        "`tf.data` pipelines, and [Estimators](./estimators.md).\n",
        "`tf.keras` makes TensorFlow easier to use without sacrificing flexibility and\n",
        "performance.\n",
        "\n",
        "To get started, import `tf.keras` as part of your TensorFlow program setup:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Qoc055N3wiUG"
      },
      "outputs": [],
      "source": [
        "!pip install pyyaml  # Required to save models in YAML format"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TgPcBFru0E1z"
      },
      "outputs": [],
      "source": [
        "import tensorflow.compat.v1 as tf\n",
        "\n",
        "from tensorflow.keras import layers\n",
        "\n",
        "print(tf.version.VERSION)\n",
        "print(tf.keras.__version__)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lj03RamP0E13"
      },
      "source": [
        "`tf.keras` can run any Keras-compatible code, but keep in mind:\n",
        "\n",
        "* The `tf.keras` version in the latest TensorFlow release might not be the same\n",
        "  as the latest `keras` version from PyPI. Check `tf.keras.__version__`.\n",
        "* When [saving a model's weights](#weights_only), `tf.keras` defaults to the\n",
        "  [checkpoint format](./checkpoints.md). Pass `save_format='h5'` to\n",
        "  use HDF5."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7e1LPcXx0gR6"
      },
      "source": [
        "## Build a simple model\n",
        "\n",
        "### Sequential model\n",
        "\n",
        "In Keras, you assemble *layers* to build *models*. A model is (usually) a graph\n",
        "of layers. The most common type of model is a stack of layers: the\n",
        "`tf.keras.Sequential` model.\n",
        "\n",
        "To build a simple, fully-connected network (i.e. multi-layer perceptron):"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "WM-DUVQB0E14"
      },
      "outputs": [],
      "source": [
        "model = tf.keras.Sequential()\n",
        "# Adds a densely-connected layer with 64 units to the model:\n",
        "model.add(layers.Dense(64, activation='relu'))\n",
        "# Add another:\n",
        "model.add(layers.Dense(64, activation='relu'))\n",
        "# Add a softmax layer with 10 output units:\n",
        "model.add(layers.Dense(10, activation='softmax'))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-ztyTipu0E18"
      },
      "source": [
        "### Configure the layers\n",
        "\n",
        "There are many `tf.keras.layers` available with some common constructor\n",
        "parameters:\n",
        "\n",
        "* `activation`: Set the activation function for the layer. This parameter is\n",
        "  specified by the name of a built-in function or as a callable object. By\n",
        "  default, no activation is applied.\n",
        "* `kernel_initializer` and `bias_initializer`: The initialization schemes\n",
        "  that create the layer's weights (kernel and bias). This parameter is a name or\n",
        "  a callable object. The kernel defaults to the `\"Glorot uniform\"` initializer, \n",
        "  and the bias defaults to zeros.\n",
        "* `kernel_regularizer` and `bias_regularizer`: The regularization schemes\n",
        "  that apply the layer's weights (kernel and bias), such as L1 or L2\n",
        "  regularization. By default, no regularization is applied.\n",
        "\n",
        "The following instantiates `tf.keras.layers.Dense` layers using constructor\n",
        "arguments:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MlL7PBtp0E19"
      },
      "outputs": [],
      "source": [
        "# Create a sigmoid layer:\n",
        "layers.Dense(64, activation='sigmoid')\n",
        "# Or:\n",
        "layers.Dense(64, activation=tf.sigmoid)\n",
        "\n",
        "# A linear layer with L1 regularization of factor 0.01 applied to the kernel matrix:\n",
        "layers.Dense(64, kernel_regularizer=tf.keras.regularizers.l1(0.01))\n",
        "\n",
        "# A linear layer with L2 regularization of factor 0.01 applied to the bias vector:\n",
        "layers.Dense(64, bias_regularizer=tf.keras.regularizers.l2(0.01))\n",
        "\n",
        "# A linear layer with a kernel initialized to a random orthogonal matrix:\n",
        "layers.Dense(64, kernel_initializer='orthogonal')\n",
        "\n",
        "# A linear layer with a bias vector initialized to 2.0s:\n",
        "layers.Dense(64, bias_initializer=tf.keras.initializers.constant(2.0))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9NR6reyk0E2A"
      },
      "source": [
        "## Train and evaluate\n",
        "\n",
        "### Set up training\n",
        "\n",
        "After the model is constructed, configure its learning process by calling the\n",
        "`compile` method:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sJ4AOn090E2A"
      },
      "outputs": [],
      "source": [
        "model = tf.keras.Sequential([\n",
        "# Adds a densely-connected layer with 64 units to the model:\n",
        "layers.Dense(64, activation='relu', input_shape=(32,)),\n",
        "# Add another:\n",
        "layers.Dense(64, activation='relu'),\n",
        "# Add a softmax layer with 10 output units:\n",
        "layers.Dense(10, activation='softmax')])\n",
        "\n",
        "model.compile(optimizer=tf.train.AdamOptimizer(0.001),\n",
        "              loss='categorical_crossentropy',\n",
        "              metrics=['accuracy'])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HG-RAa9F0E2D"
      },
      "source": [
        "`tf.keras.Model.compile` takes three important arguments:\n",
        "\n",
        "* `optimizer`: This object specifies the training procedure. Pass it optimizer\n",
        "  instances from the `tf.train` module, such as\n",
        "  `tf.train.AdamOptimizer`, `tf.train.RMSPropOptimizer`, or\n",
        "  `tf.train.GradientDescentOptimizer`.\n",
        "* `loss`: The function to minimize during optimization. Common choices include\n",
        "  mean square error (`mse`), `categorical_crossentropy`, and\n",
        "  `binary_crossentropy`. Loss functions are specified by name or by\n",
        "  passing a callable object from the `tf.keras.losses` module.\n",
        "* `metrics`: Used to monitor training. These are string names or callables from\n",
        "  the `tf.keras.metrics` module.\n",
        "\n",
        "The following shows a few examples of configuring a model for training:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "St4Mgdar0E2E"
      },
      "outputs": [],
      "source": [
        "# Configure a model for mean-squared error regression.\n",
        "model.compile(optimizer=tf.train.AdamOptimizer(0.01),\n",
        "              loss='mse',       # mean squared error\n",
        "              metrics=['mae'])  # mean absolute error\n",
        "\n",
        "# Configure a model for categorical classification.\n",
        "model.compile(optimizer=tf.train.RMSPropOptimizer(0.01),\n",
        "              loss=tf.keras.losses.categorical_crossentropy,\n",
        "              metrics=[tf.keras.metrics.categorical_accuracy])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yjI5rbi80E2G"
      },
      "source": [
        "### Input NumPy data\n",
        "\n",
        "For small datasets, use in-memory [NumPy](https://www.numpy.org/)\n",
        "arrays to train and evaluate a model. The model is \"fit\" to the training data\n",
        "using the `fit` method:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3CvP6L-m0E2I"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "\n",
        "def random_one_hot_labels(shape):\n",
        "  n, n_class = shape\n",
        "  classes = np.random.randint(0, n_class, n)\n",
        "  labels = np.zeros((n, n_class))\n",
        "  labels[np.arange(n), classes] = 1\n",
        "  return labels\n",
        "\n",
        "data = np.random.random((1000, 32))\n",
        "labels = random_one_hot_labels((1000, 10))\n",
        "\n",
        "model.fit(data, labels, epochs=10, batch_size=32)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N-pnVaFe0E2N"
      },
      "source": [
        "`tf.keras.Model.fit` takes three important arguments:\n",
        "\n",
        "* `epochs`: Training is structured into *epochs*. An epoch is one iteration over\n",
        "  the entire input data (this is done in smaller batches).\n",
        "* `batch_size`: When passed NumPy data, the model slices the data into smaller\n",
        "  batches and iterates over these batches during training. This integer\n",
        "  specifies the size of each batch. Be aware that the last batch may be smaller\n",
        "  if the total number of samples is not divisible by the batch size.\n",
        "* `validation_data`: When prototyping a model, you want to easily monitor its\n",
        "  performance on some validation data. Passing this argument—a tuple of inputs\n",
        "  and labels—allows the model to display the loss and metrics in inference mode\n",
        "  for the passed data, at the end of each epoch.\n",
        "\n",
        "Here's an example using `validation_data`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "gFcXzVQa0E2N"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "\n",
        "data = np.random.random((1000, 32))\n",
        "labels = random_one_hot_labels((1000, 10))\n",
        "\n",
        "val_data = np.random.random((100, 32))\n",
        "val_labels = random_one_hot_labels((100, 10))\n",
        "\n",
        "model.fit(data, labels, epochs=10, batch_size=32,\n",
        "          validation_data=(val_data, val_labels))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-6ImyXzz0E2Q"
      },
      "source": [
        "### Input tf.data datasets\n",
        "\n",
        "Use the [Datasets API](./datasets.md) to scale to large datasets\n",
        "or multi-device training. Pass a `tf.data.Dataset` instance to the `fit`\n",
        "method:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OziqhpIj0E2R"
      },
      "outputs": [],
      "source": [
        "# Instantiates a toy dataset instance:\n",
        "dataset = tf.data.Dataset.from_tensor_slices((data, labels))\n",
        "dataset = dataset.batch(32)\n",
        "dataset = dataset.repeat()\n",
        "\n",
        "# Don't forget to specify `steps_per_epoch` when calling `fit` on a dataset.\n",
        "model.fit(dataset, epochs=10, steps_per_epoch=30)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I7BcMHkB0E2U"
      },
      "source": [
        "Here, the `fit` method uses the `steps_per_epoch` argument—this is the number of\n",
        "training steps the model runs before it moves to the next epoch. Since the\n",
        "`Dataset` yields batches of data, this snippet does not require a `batch_size`.\n",
        "\n",
        "Datasets can also be used for validation:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YPMb3A0N0E2V"
      },
      "outputs": [],
      "source": [
        "dataset = tf.data.Dataset.from_tensor_slices((data, labels))\n",
        "dataset = dataset.batch(32).repeat()\n",
        "\n",
        "val_dataset = tf.data.Dataset.from_tensor_slices((val_data, val_labels))\n",
        "val_dataset = val_dataset.batch(32).repeat()\n",
        "\n",
        "model.fit(dataset, epochs=10, steps_per_epoch=30,\n",
        "          validation_data=val_dataset,\n",
        "          validation_steps=3)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IgGdlXso0E2X"
      },
      "source": [
        "### Evaluate and predict\n",
        "\n",
        "The `tf.keras.Model.evaluate` and `tf.keras.Model.predict` methods can use NumPy\n",
        "data and a `tf.data.Dataset`.\n",
        "\n",
        "To *evaluate* the inference-mode loss and metrics for the data provided:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mhDbOHEK0E2Y"
      },
      "outputs": [],
      "source": [
        "data = np.random.random((1000, 32))\n",
        "labels = random_one_hot_labels((1000, 10))\n",
        "\n",
        "model.evaluate(data, labels, batch_size=32)\n",
        "\n",
        "model.evaluate(dataset, steps=30)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UXUTmDfb0E2b"
      },
      "source": [
        "And to *predict* the output of the last layer in inference for the data provided,\n",
        "as a NumPy array:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9e3JsSoQ0E2c"
      },
      "outputs": [],
      "source": [
        "result = model.predict(data, batch_size=32)\n",
        "print(result.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fzEOW4Cn0E2h"
      },
      "source": [
        "## Build advanced models\n",
        "\n",
        "### Functional API\n",
        "\n",
        " The `tf.keras.Sequential` model is a simple stack of layers that cannot\n",
        "represent arbitrary models. Use the\n",
        "[Keras functional API](https://keras.io/getting-started/functional-api-guide/)\n",
        "to build complex model topologies such as:\n",
        "\n",
        "* Multi-input models,\n",
        "* Multi-output models,\n",
        "* Models with shared layers (the same layer called several times),\n",
        "* Models with non-sequential data flows (e.g. residual connections).\n",
        "\n",
        "Building a model with the functional API works like this:\n",
        "\n",
        "1. A layer instance is callable and returns a tensor.\n",
        "2. Input tensors and output tensors are used to define a `tf.keras.Model`\n",
        "   instance.\n",
        "3. This model is trained just like the `Sequential` model.\n",
        "\n",
        "The following example uses the functional API to build a simple, fully-connected\n",
        "network:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mROj832r0E2i"
      },
      "outputs": [],
      "source": [
        "inputs = tf.keras.Input(shape=(32,))  # Returns a placeholder tensor\n",
        "\n",
        "# A layer instance is callable on a tensor, and returns a tensor.\n",
        "x = layers.Dense(64, activation='relu')(inputs)\n",
        "x = layers.Dense(64, activation='relu')(x)\n",
        "predictions = layers.Dense(10, activation='softmax')(x)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AFmspHeG1_W7"
      },
      "source": [
        "Instantiate the model given inputs and outputs."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5k5uzlyu16HM"
      },
      "outputs": [],
      "source": [
        "model = tf.keras.Model(inputs=inputs, outputs=predictions)\n",
        "\n",
        "# The compile step specifies the training configuration.\n",
        "model.compile(optimizer=tf.train.RMSPropOptimizer(0.001),\n",
        "              loss='categorical_crossentropy',\n",
        "              metrics=['accuracy'])\n",
        "\n",
        "# Trains for 5 epochs\n",
        "model.fit(data, labels, batch_size=32, epochs=5)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EcKSLH3i0E2k"
      },
      "source": [
        "### Model subclassing\n",
        "\n",
        "Build a fully-customizable model by subclassing `tf.keras.Model` and defining\n",
        "your own forward pass. Create layers in the `__init__` method and set them as\n",
        "attributes of the class instance. Define the forward pass in the `call` method.\n",
        "\n",
        "Model subclassing is particularly useful when\n",
        "[eager execution](./eager.ipynb) is enabled since the forward pass\n",
        "can be written imperatively.\n",
        "\n",
        "Key Point: Use the right API for the job. While model subclassing offers\n",
        "flexibility, it comes at a cost of greater complexity and more opportunities for\n",
        "user errors. If possible, prefer the functional API.\n",
        "\n",
        "The following example shows a subclassed `tf.keras.Model` using a custom forward\n",
        "pass:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KLiHWzcn2Fzk"
      },
      "outputs": [],
      "source": [
        "class MyModel(tf.keras.Model):\n",
        "\n",
        "  def __init__(self, num_classes=10):\n",
        "    super(MyModel, self).__init__(name='my_model')\n",
        "    self.num_classes = num_classes\n",
        "    # Define your layers here.\n",
        "    self.dense_1 = layers.Dense(32, activation='relu')\n",
        "    self.dense_2 = layers.Dense(num_classes, activation='sigmoid')\n",
        "\n",
        "  def call(self, inputs):\n",
        "    # Define your forward pass here,\n",
        "    # using layers you previously defined (in `__init__`).\n",
        "    x = self.dense_1(inputs)\n",
        "    return self.dense_2(x)\n",
        "\n",
        "  def compute_output_shape(self, input_shape):\n",
        "    # You need to override this function if you want to use the subclassed model\n",
        "    # as part of a functional-style model.\n",
        "    # Otherwise, this method is optional.\n",
        "    shape = tf.TensorShape(input_shape).as_list()\n",
        "    shape[-1] = self.num_classes\n",
        "    return tf.TensorShape(shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ShDD4fv72KGc"
      },
      "source": [
        "Instantiate the new model class:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "42C-qQHm0E2l"
      },
      "outputs": [],
      "source": [
        "model = MyModel(num_classes=10)\n",
        "\n",
        "# The compile step specifies the training configuration.\n",
        "model.compile(optimizer=tf.train.RMSPropOptimizer(0.001),\n",
        "              loss='categorical_crossentropy',\n",
        "              metrics=['accuracy'])\n",
        "\n",
        "# Trains for 5 epochs.\n",
        "model.fit(data, labels, batch_size=32, epochs=5)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yqRQiKj20E2o"
      },
      "source": [
        "### Custom layers\n",
        "\n",
        "Create a custom layer by subclassing `tf.keras.layers.Layer` and implementing\n",
        "the following methods:\n",
        "\n",
        "* `build`: Create the weights of the layer. Add weights with the `add_weight`\n",
        "  method.\n",
        "* `call`: Define the forward pass.\n",
        "* `compute_output_shape`: Specify how to compute the output shape of the layer\n",
        "  given the input shape.\n",
        "* Optionally, a layer can be serialized by implementing the `get_config` method\n",
        "  and the `from_config` class method.\n",
        "\n",
        "Here's an example of a custom layer that implements a `matmul` of an input with\n",
        "a kernel matrix:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "l7BFnIHr2WNc"
      },
      "outputs": [],
      "source": [
        "class MyLayer(layers.Layer):\n",
        "\n",
        "  def __init__(self, output_dim, **kwargs):\n",
        "    self.output_dim = output_dim\n",
        "    super(MyLayer, self).__init__(**kwargs)\n",
        "\n",
        "  def build(self, input_shape):\n",
        "    shape = tf.TensorShape((input_shape[1], self.output_dim))\n",
        "    # Create a trainable weight variable for this layer.\n",
        "    self.kernel = self.add_weight(name='kernel',\n",
        "                                  shape=shape,\n",
        "                                  initializer='uniform',\n",
        "                                  trainable=True)\n",
        "    # Make sure to call the `build` method at the end\n",
        "    super(MyLayer, self).build(input_shape)\n",
        "\n",
        "  def call(self, inputs):\n",
        "    return tf.matmul(inputs, self.kernel)\n",
        "\n",
        "  def compute_output_shape(self, input_shape):\n",
        "    shape = tf.TensorShape(input_shape).as_list()\n",
        "    shape[-1] = self.output_dim\n",
        "    return tf.TensorShape(shape)\n",
        "\n",
        "  def get_config(self):\n",
        "    base_config = super(MyLayer, self).get_config()\n",
        "    base_config['output_dim'] = self.output_dim\n",
        "    return base_config\n",
        "\n",
        "  @classmethod\n",
        "  def from_config(cls, config):\n",
        "    return cls(**config)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8wXDRgXV2ZrF"
      },
      "source": [
        "Create a model using your custom layer:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uqH-cY0h0E2p"
      },
      "outputs": [],
      "source": [
        "model = tf.keras.Sequential([\n",
        "    MyLayer(10),\n",
        "    layers.Activation('softmax')])\n",
        "\n",
        "# The compile step specifies the training configuration\n",
        "model.compile(optimizer=tf.train.RMSPropOptimizer(0.001),\n",
        "              loss='categorical_crossentropy',\n",
        "              metrics=['accuracy'])\n",
        "\n",
        "# Trains for 5 epochs.\n",
        "model.fit(data, labels, batch_size=32, epochs=5)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Lu8cc3AJ0E2v"
      },
      "source": [
        "## Callbacks\n",
        "\n",
        "A callback is an object passed to a model to customize and extend its behavior\n",
        "during training. You can write your own custom callback, or use the built-in\n",
        "`tf.keras.callbacks` that include:\n",
        "\n",
        "* `tf.keras.callbacks.ModelCheckpoint`: Save checkpoints of your model at\n",
        "  regular intervals.\n",
        "* `tf.keras.callbacks.LearningRateScheduler`: Dynamically change the learning\n",
        "  rate.\n",
        "* `tf.keras.callbacks.EarlyStopping`: Interrupt training when validation\n",
        "  performance has stopped improving.\n",
        "* `tf.keras.callbacks.TensorBoard`: Monitor the model's behavior using\n",
        "  [TensorBoard](https://tensorflow.org/tensorboard).\n",
        "\n",
        "To use a `tf.keras.callbacks.Callback`, pass it to the model's `fit` method:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rdYwzSYV0E2v"
      },
      "outputs": [],
      "source": [
        "callbacks = [\n",
        "  # Interrupt training if `val_loss` stops improving for over 2 epochs\n",
        "  tf.keras.callbacks.EarlyStopping(patience=2, monitor='val_loss'),\n",
        "  # Write TensorBoard logs to `./logs` directory\n",
        "  tf.keras.callbacks.TensorBoard(log_dir='./logs')\n",
        "]\n",
        "model.fit(data, labels, batch_size=32, epochs=5, callbacks=callbacks,\n",
        "          validation_data=(val_data, val_labels))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ghhaGfX62abv"
      },
      "source": [
        "<a name='save_and_restore'></a>\n",
        "## Save and restore"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qnl7K-aI0E2z"
      },
      "source": [
        "<a name='weights_only'></a>\n",
        "### Weights only\n",
        "\n",
        "Save and load the weights of a model using `tf.keras.Model.save_weights`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uQIANjB94fLB"
      },
      "outputs": [],
      "source": [
        "model = tf.keras.Sequential([\n",
        "layers.Dense(64, activation='relu', input_shape=(32,)),\n",
        "layers.Dense(10, activation='softmax')])\n",
        "\n",
        "model.compile(optimizer=tf.train.AdamOptimizer(0.001),\n",
        "              loss='categorical_crossentropy',\n",
        "              metrics=['accuracy'])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4eoHJ-ny0E21"
      },
      "outputs": [],
      "source": [
        "# Save weights to a TensorFlow Checkpoint file\n",
        "model.save_weights('./weights/my_model')\n",
        "\n",
        "# Restore the model's state,\n",
        "# this requires a model with the same architecture.\n",
        "model.load_weights('./weights/my_model')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u25Id3xe0E25"
      },
      "source": [
        "By default, this saves the model's weights in the\n",
        "[TensorFlow checkpoint](./checkpoints.md) file format. Weights can\n",
        "also be saved to the Keras HDF5 format (the default for the multi-backend\n",
        "implementation of Keras):"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JSAYoFEd0E26"
      },
      "outputs": [],
      "source": [
        "# Save weights to a HDF5 file\n",
        "model.save_weights('my_model.h5', save_format='h5')\n",
        "\n",
        "# Restore the model's state\n",
        "model.load_weights('my_model.h5')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mje_yKL10E29"
      },
      "source": [
        "### Configuration only\n",
        "\n",
        "A model's configuration can be saved—this serializes the model architecture\n",
        "without any weights. A saved configuration can recreate and initialize the same\n",
        "model, even without the code that defined the original model. Keras supports the\n",
        "JSON serialization format:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "EbET0oJTzGkq"
      },
      "outputs": [],
      "source": [
        "# Serialize a model to JSON format\n",
        "json_string = model.to_json()\n",
        "json_string"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pX_badhH3yWV"
      },
      "outputs": [],
      "source": [
        "import json\n",
        "import pprint\n",
        "pprint.pprint(json.loads(json_string))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q7CIa05r4yTb"
      },
      "source": [
        "Recreate the model (newly initialized) from the JSON:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "J9UFv9k00E2_"
      },
      "outputs": [],
      "source": [
        "fresh_model = tf.keras.models.model_from_json(json_string)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xPvOSSzM0E3B"
      },
      "source": [
        "Caution: Subclassed models are not serializable because their architecture is\n",
        "defined by the Python code in the body of the `call` method."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iu8qMwld4-71"
      },
      "source": [
        "### Entire model\n",
        "\n",
        "The entire model can be saved to a file that contains the weight values, the\n",
        "model's configuration, and even the optimizer's configuration. This allows you\n",
        "to checkpoint a model and resume training later—from the exact same\n",
        "state—without access to the original code."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "45oNY34Z0E3C"
      },
      "outputs": [],
      "source": [
        "# Create a trivial model\n",
        "model = tf.keras.Sequential([\n",
        "  layers.Dense(64, activation='relu', input_shape=(32,)),\n",
        "  layers.Dense(10, activation='softmax')\n",
        "])\n",
        "model.compile(optimizer='rmsprop',\n",
        "              loss='categorical_crossentropy',\n",
        "              metrics=['accuracy'])\n",
        "model.fit(data, labels, batch_size=32, epochs=5)\n",
        "\n",
        "\n",
        "# Save entire model to a HDF5 file\n",
        "model.save('my_model.h5')\n",
        "\n",
        "# Recreate the exact same model, including weights and optimizer.\n",
        "model = tf.keras.models.load_model('my_model.h5')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PMOWhDOB0E3E"
      },
      "source": [
        "<a name='eager_execution'></a>\n",
        "## Eager execution\n",
        "\n",
        "[Eager execution](./eager.ipynb) is an imperative programming\n",
        "environment that evaluates operations immediately. This is not required for\n",
        "Keras, but is supported by `tf.keras` and useful for inspecting your program and\n",
        "debugging.\n",
        "\n",
        "All of the `tf.keras` model-building APIs are compatible with eager execution.\n",
        "And while the `Sequential` and functional APIs can be used, eager execution\n",
        "especially benefits *model subclassing* and building *custom layers*—the APIs\n",
        "that require you to write the forward pass as code (instead of the APIs that\n",
        "create models by assembling existing layers).\n",
        "\n",
        "See the [eager execution guide](./eager.ipynb#build_a_model) for\n",
        "examples of using Keras models with custom training loops and `tf.GradientTape`."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2wG3NVco5B5V"
      },
      "source": [
        "## Distribution\n",
        "\n",
        "### Estimators\n",
        "\n",
        "The [Estimators](./estimators.md) API is used for training models\n",
        "for distributed environments. This targets industry use cases such as\n",
        "distributed training on large datasets that can export a model for production.\n",
        "\n",
        "A `tf.keras.Model` can be trained with the `tf.estimator` API by converting the\n",
        "model to an `tf.estimator.Estimator` object with\n",
        "`tf.keras.estimator.model_to_estimator`. See\n",
        "[Creating Estimators from Keras models](./estimators.md#creating-estimators-from-keras-models)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cVg0vfTO0E3E"
      },
      "outputs": [],
      "source": [
        "model = tf.keras.Sequential([layers.Dense(64, activation='relu', input_shape=(32,)),\n",
        "                          layers.Dense(10,activation='softmax')])\n",
        "\n",
        "model.compile(optimizer=tf.train.RMSPropOptimizer(0.001),\n",
        "              loss='categorical_crossentropy',\n",
        "              metrics=['accuracy'])\n",
        "\n",
        "estimator = tf.keras.estimator.model_to_estimator(model)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S7FKvikO0E3H"
      },
      "source": [
        "Note: Enable [eager execution](./eager.ipynb) for debugging\n",
        "[Estimator input functions](./premade_estimators.md#create-input-functions)\n",
        "and inspecting data."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6PJZ6e9J5JHF"
      },
      "source": [
        "### Multiple GPUs\n",
        "\n",
        "`tf.keras` models can run on multiple GPUs using\n",
        "`tf.distribute.DistributionStrategy`. This API provides distributed\n",
        "training on multiple GPUs with almost no changes to existing code.\n",
        "\n",
        "Currently, `tf.distribute.MirroredStrategy` is the only supported\n",
        "distribution strategy. `MirroredStrategy` does in-graph replication with\n",
        "synchronous training using all-reduce on a single machine. To use\n",
        "`DistributionStrategy` with Keras, convert the `tf.keras.Model` to a\n",
        "`tf.estimator.Estimator` with `tf.keras.estimator.model_to_estimator`, then\n",
        "train the estimator\n",
        "\n",
        "The following example distributes a `tf.keras.Model` across multiple GPUs on a\n",
        "single machine.\n",
        "\n",
        "First, define a simple model:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sbaRr7g-0E3I"
      },
      "outputs": [],
      "source": [
        "model = tf.keras.Sequential()\n",
        "model.add(layers.Dense(16, activation='relu', input_shape=(10,)))\n",
        "model.add(layers.Dense(1, activation='sigmoid'))\n",
        "\n",
        "optimizer = tf.train.GradientDescentOptimizer(0.2)\n",
        "\n",
        "model.compile(loss='binary_crossentropy', optimizer=optimizer)\n",
        "model.summary()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yw4hSJme0E3L"
      },
      "source": [
        "Define an *input pipeline*. The `input_fn` returns a `tf.data.Dataset` object\n",
        "used to distribute the data across multiple devices—with each device processing\n",
        "a slice of the input batch."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CxJW1QMH0E3L"
      },
      "outputs": [],
      "source": [
        "def input_fn():\n",
        "  x = np.random.random((1024, 10))\n",
        "  y = np.random.randint(2, size=(1024, 1))\n",
        "  x = tf.cast(x, tf.float32)\n",
        "  dataset = tf.data.Dataset.from_tensor_slices((x, y))\n",
        "  dataset = dataset.repeat(10)\n",
        "  dataset = dataset.batch(32)\n",
        "  return dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rO9MiL6X0E3O"
      },
      "source": [
        "Next, create a `tf.estimator.RunConfig` and set the `train_distribute` argument\n",
        "to the `tf.distribute.MirroredStrategy` instance. When creating\n",
        "`MirroredStrategy`, you can specify a list of devices or set the `num_gpus`\n",
        "argument. The default uses all available GPUs, like the following:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "BEwFq4PM0E3P"
      },
      "outputs": [],
      "source": [
        "strategy = tf.distribute.MirroredStrategy()\n",
        "config = tf.estimator.RunConfig(train_distribute=strategy)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TcnwYVun0E3R"
      },
      "source": [
        "Convert the Keras model to a `tf.estimator.Estimator` instance:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VSvbuIID0E3S"
      },
      "outputs": [],
      "source": [
        "keras_estimator = tf.keras.estimator.model_to_estimator(\n",
        "  keras_model=model,\n",
        "  config=config,\n",
        "  model_dir='/tmp/model_dir')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N6BXU5F90E3U"
      },
      "source": [
        "Finally, train the `Estimator` instance by providing the `input_fn` and `steps`\n",
        "arguments:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XKoJ2wUH0E3U"
      },
      "outputs": [],
      "source": [
        "keras_estimator.train(input_fn=input_fn, steps=10)"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "keras.ipynb",
            "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
